Control of Limb Device

ABSTRACT

The invention refers to the area of control of a limb device in the form of an artificial limb for a human or a robot limb. In particular, the invention is related to a control unit for electrically controlling an electrically controllable limb device, the limb device comprising a plurality of actuators, the control unit comprising a first interface for connecting the control unit to the limb device, the control unit comprising a second interface for connecting the control unit to a data gathering device comprising one or more sensing devices, the control unit comprising a processing unit which is arranged for controlling the limb device at least based on data gathered by the data gathering device, wherein the control unit is arranged for outputting one single control action step to the actuators of the limb device calculated by the processing unit based on a first data or data combination received from the data gathering device, and the control unit is arranged for outputting a plurality of control action steps to the actuators of the limb device calculated by the processing unit based on a second data or data combination received from the data gathering device, the second data or data combination being different from the first data or data combination, the plurality of control action steps inducing a more complex automatic movement of the limb device that the one single control action step. The invention further refers to a system comprising such a control unit, a method for controlling an electrically controllable limb device and a computer program.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of application Ser. No. 14/303,229filed Jun. 12, 2014, by Dario Farina et al. for “Control for LimbDevice”. This application claims the benefit of the filing date ofapplication Ser. No. 14/303,229.

BACKGROUND OF THE INVENTION

Field of the Invention

The invention refers to the area of control of a limb device in the formof an artificial limb for a human or a robot limb. In particular, theinvention is related to a control unit for electrically controlling anelectrically controllable limb device, a system comprising such acontrol unit, a method for controlling an electrically controllable limbdevice and a computer program.

While the invention is fully applicable both to the control ofartificial limbs for humans and for limbs of robots, in the followingdescription it is mainly referred to the area of artificial limbs forhumans, e.g. an arm or hand prosthesis. The limb can be e.g. anyextremity, like an arm or a leg.

Description

In prior art devices, myoelectric control is used for controlling anelectrically controllable limb device. Myoelectric control is the mostcommon method in the literature, and it is in fact the only method thatwas accepted by the industry and implemented in commercial products. Inmyoelectric control, muscle activity is assessed by usingelectromyography (EMG) recordings and the acquired EMG signals aredecoded to reflect the intention of the user. The reasons for usingmyoelectric signals are subconscious control and simple interface. Thiscontrol method has been in the focus of research already for decades,but it is now evident that there is a large gap between what has beendeveloped and tested in academia, and what has been actually implementedin the working, commercial devices. Basically, only the simplest form ofmyoelectric control (i.e., two-channel/two-command interface) showed tobe robust and practical enough to be applied outside of the researchlab. The limitations of this approach are even more evident, as more andmore modern, sophisticated prostheses are being designed and launched tothe market, e.g., Otto Bock Michelangelo Hand, Touch Bionics i-Limb,RSLSteeper Bebionic Hand. These systems are complex mechanicalmechanisms with several independently controllable degrees of freedom.Controlling such a flexible functionality represents a challenging taskthat is hard to integrate into the classic pattern based, master-slavescenario, in which the user has to control all the aspects of theprosthesis during the execution of a task.

With an amputation, a person suffers also a loss of perception (sensoryfunctions). The current prosthesis can restore the motor functionalityto a certain degree, but none of the existing devices offers any kind ofsoma-somatosensory and proprioceptive feedback, e.g. touch, force, jointpositions etc. The idea to close the loop and provide the feedback fromthe prosthesis to the user has been presented long ago. The lostsensations can be restored using sensory substitution. In this method,the information that was originally received by the lost limb isdelivered by stimulating an alternative, still intact sense. Forexample, the prosthesis can be equipped with the sensors measuring thecontact force and joint angles, and the information from these sensorscan be delivered by stimulating the skin of the residual limb. The mostcommon methods for the activation of the tactile sense are electricaland direct mechanical stimulation, e.g. by miniature vibratory motors.Although these methods are relatively easy to implement, the resultingfeedback channel has limited information transfer due to the inherentlimitations of the artificially stimulated tactile sense.

SUMMARY OF THE INVENTION

It is an object of the invention to overcome the aforementioneddisadvantages.

The object of the invention is achieved by a control unit forelectrically controlling an electrically controllable limb device in theform of an artificial limb for a human or a robot limb, the limb devicecomprising a plurality of actuators, the control unit comprising a firstinterface for connecting the control unit to the limb device, thecontrol unit comprising a second interface for connecting the controlunit to a data gathering device comprising one or more sensing devices,the control unit comprising a processing unit which is arranged forcontrolling the limb device at least based on data gathered by the datagathering device, wherein the control unit is arranged for outputtingone single control action step to the actua-actuators of the limb devicecalculated by the processing unit based on a first data or datacombination received from the data gathering device, and the controlunit is arranged for outputting a plurality of control action steps tothe actuators of the limb device calculated by the processing unit basedon a second data or data combination received from the data gatheringdevice, the second data or data combination being different from thefirst data or data combination, the plurality of control action stepsinducing a more complex automatic movement of the limb device that theone single control action step.

The invention has the advantage that the control unit which acts as anartificial controller is made more “intelligent” so that it is able toadd autonomous decisions to the recognised intentions of the user andtherefore automatically do a plurality of control action steps. Thisrequires less involvement of the user. In this way, a conceptual shiftcompared to prior art concepts is proposed.

The plurality of action steps which are automatically performed by thecontrol unit can be performed simultaneously, which means at the sametime, or in a sequential manner. Also, a combination of somesimultaneously performed control action steps and some sequentiallyperformed control action steps of the plurality of control action stepscan be implemented advantageously.

As explained above, a critical assessment of the current state of theart in the control of active prostheses has shown that the currentcontrol methods have serious limitations, especially when the system tobe controlled is complex (modern prostheses). For example, advanced,contemporary hand prostheses implement several grasp types such aslateral, palmar, or pinch grip. If the wrist or elbow is alsocontrollable as in, for example, Dynamic Arm from Otto Bock, the arm andhand orientation have to be adjusted as well. The optimal grasp toemploy depends on the features of the target object, e.g., palmar graspfor a bottle, and lateral grip for a key. With a conventionalmyoelectric control, the user operating the prosthesis has to implementseveral sequential steps each time he/she wants to grasp an object: 1)analyze the object and decide the best grasp type, 2) generate thesignals for the selection of a certain grasp type (e.g., several briefmuscle contractions), 3) adjust the opening of the hand so that thetarget object can fit (e.g., several longer muscle contractions), andfinally 4) command the hand to close (e.g., one long musclecontraction). If in addition the wrist and elbow joints need to beadjusted, the user has to switch to joint control mode by co-contractingthe muscles and then generate the signals to move the joints. Therefore,the user has to go through a sequence of steps, one by one, controllingdifferent aspects of the prosthetic system. Of course, this process canbe complicated, long and difficult for the user. He/she has to rememberthe sequence, generate the proper command signals, change modes (e.g.,from grasp selection to hand closing/opening) and remember the availableoptions (e.g., how many grasp types and in which order). This processbecomes more complicated for the user as the flexibility of the deviceincreases by the introduction of more grasp types, or more controllablejoints. These disadvantages can be overcome by the present invention.

According to an advantageous embodiment of the invention, the controlunit is arranged for receiving data via the second interface from thedata gathering device which comprises at least 3D (three dimensional)information including depth information, in particular 3D informationfrom surrounding areas of the limb device. With such 3D informationincluding depth information the abilities of the user to control thelimb device are further optimized. The 3D information including depthinformation can be provided by the mentioned stereovision cameraarrangement or any other type of sensor which is able to scan theambient areas and generate such information, like a laser scanner, radaror ultrasonic device.

According to an advantageous embodiment of the invention, the controlunit is arranged for receiving data via the second interface from thedata gathering device which comprises at least one of camera data,3D-scanning device data such as stereovision camera arrangement data,laser scanning data, white light scanning data, and inertial sensingunit data. This has the advantage that additional sensor data forautomatically controlling the limb device are available. In prior artdevices, the lack of feedback from the prosthesis is a drawback for theusers, affecting the performance and acceptance of the system. Since theusers do not know the positions of the joints, they have to monitor theprosthesis visually in addition to the object that they are about tograsp, and this is a cognitive burden. Once they have grasped theobject, they do not have explicit information about the generatedcontact force, and this can lead to a breakage or slippage of themanipulated object. These drawbacks are overcome by the presentinvention. With the additional data it is possible to performautomatically several types of grasp motions, particularly in a moresensitive way than with prior art devices. For example, it is possibleto perform some control action steps fully automatically, like selectionof grasp type and size, and orientation of the arm and/or hand of thelimb device without the need for the user involvement. As a result, thecontrol is simplified and the cognitive burden to the user is minimised.In addition, the control loop is now closed by providing enrichedinformation about the current state of the limb device to the user. Inparticular, the enriched in-information can contain visual informationfrom the camera and/or the 3D-scanning device. The inertial sensing unitmay comprise one or more of accelerometers, gyroscopes andmagnetometers.

The 3D-scanning device can be e.g. a stereovision camera arrangement, alaser scanning device, e.g. with pulse shift, time of flight etc., aradar device, a white light scanning device.

According to an advantageous embodiment of the invention, the controlunit comprises a third interface for connecting the control unit to avisual output device to be used by a user of the limb device, whereinthe control unit is arranged for outputting visual data to the visualoutput device based on data received from the data gathering deviceand/or control steps outputted to the limb device. This has theadvantage that information can be transferred to the user by visualmeans which allows for high information bandwidth in recognising ofinformation by the user.

According to an advantageous embodiment of the invention, the visualdevice is a stereovision augmented reality device which is arranged forbeing worn by a user of the limb device, allowing to output visualinformation to the user which comprises at least 3D informationincluding depth information. This has the advantage that the visualinformation to be presented to the user can be further enriched by usingaugmented reality and stereovision. The use of augmented reality has theadvantage that the user still has visual contact with the real world,but can be supported automatically by e.g. highlighting certain items onthe visual output device with a direct relationship to the real item.

According to an advantageous embodiment of the invention, the controlunit is arranged for outputting visual data to the visual output deviceshowing positions of the joints of the limb device. This has theadvantage that the user can visually control the operation of the limbdevice in an improved manner by using additional visual informationcompared to the usual, unsupported direct view on the limb device.

According to an advantageous embodiment of the invention, the controlunit is arranged for outputting a marking signal on the visual outputdevice based upon an internal selection of an object which is to begrasped by the limb device, the marking signal marking the selectedobject on the visual output device. This has the advantage that the usercan use the limb device in a more spontaneous and intuitive way thanprior art devices.

According to an advantageous embodiment of the invention, the controlunit is arranged for calculating a 3D (three dimensional) orientation ofthe limb device or a part of it from the data received from the datagathering device and for outputting visual information about thecalculated 3D orientation on the visual output device. This has theadvantage that the user gets through the visual output device anadditional synthetic view on the limb device, wherein the viewingdirection can be chosen by the user, without a need for repositioning acamera.

According to an advantageous embodiment of the invention, the controlunit comprises another interface for connecting the control unit to afeedback device providing feedback to a user of the limb device, whereinthe control unit is arranged for outputting feedback data to thefeedback device based on data received from the data gathering deviceand/or control steps outputted to the limb device. Through the feedbackdevice additional or alternative feedback to the visual feedback can begiven to the user. This has the advantage that the feedback given to theuser can be further enriched. The feedback device can be at least one ofa force feedback device, a tactile feedback device and a temperaturefeedback device.

According to another object of the invention, there is provided a systemcomprising a control unit, a data gathering device comprising one ormore sensing devices and an electrically controllable limb device in theform of an artificial limb for a human or a robot limb, the limb devicecomprising a plurality of actuators, the control unit electricallycontrolling the limb device via the first interface based upon datareceived via the second interface from the data gathering device. Thesystem comprises the aforementioned advantages over prior art systems.

According to an advantageous embodiment of the invention, the datagathering device comprises at least one of camera, 3D-scanning devicesuch as stereovision camera arrangement, laser scanning, white lightscanning, and inertial sensing unit. This has the advantage thatadditional sensor data for automatically controlling the limb device areavailable. In prior art devices, the lack of feedback from theprosthesis is a drawback for the users, affecting the performance andacceptance of the system. Since the users do not know the positions ofthe joints, they have to monitor the prosthesis visually in addition tothe object that they are about to grasp, and this is a cognitive burden.Once they have grasped the object, they do not have explicit informationabout the generated contact force, and this can lead to a breakage orslippage of the manipulated object. These drawbacks are overcome by thepresent invention. With the additional data it is possible to performautomatically several types of grasp motions, particularly in a moresensitive way than with prior art devices. For example, it is possibleto perform some control action steps fully automatically, like selectionof grasp type and size, and orientation of the arm and/or hand of thelimb device without the need for the user involvement. As a result, thecontrol is simplified and the cognitive burden to the user is minimised.In addition, the control loop is now closed by providing enrichedinformation about the current state of the limb device to the user. Inparticular, the enriched information can contain visual information fromthe camera and/or the 3D-scanning device. The inertial sensing unit maycomprise one or more of accelerometers, gyroscopes and magnetometers.

According to an advantageous embodiment of the invention, the datagathering device comprises a 3D data gathering device outputting atleast 3D information including depth information.

According to an advantageous embodiment of the invention, at least oneof a camera device, an ultrasonic distance sensor, a laser pointer and alaser scanner is mounted on the limb device. This has the advantage thatthe control of the limb device can be improved through additionalsensing and pointing devices. For example, by means of the camera devicemounted on the limb device an additional view on objects to be graspedcan be given to the user. By an ultrasonic distance sensor theapproaching process of the limb device to an object to be grasped can becontrolled more efficiently and be made smoother. The laser pointer canbe used to point to items to be grasped by the limb device in order tohighlight them in the real world. Further, a laser scanner can be usedfor optimising movement of the limb device to an item to be grasped.

According to an advantageous embodiment of the invention, the systemcomprises a visual output device which is connected to the control unitthrough its third interface. This has the advantage that the user can besupported with visual information through the visual output device.

According to an advantageous embodiment of the invention, the visualoutput device comprises at least one a display or a stereodisplay, acamera or a stereovision camera arrangement. This has the advantage thatthe system, in particular the control unit, can control whetherinternally calculated visual information (virtual information) orinformation from the real world (real information) or a mixture thereofis visualised through the visual output device. In such case the realinformation is captured by the camera or stereovision cameraarrangement.

According to an advantageous embodiment of the invention, the systemcomprises a feedback device which is connected to the control unitthrough another interface of the control unit, the feedback deviceproviding feedback to a user of the limb device controlled by thecontrol unit based on data received from the data gathering deviceand/or control steps outputted to the limb device. The feedback devicecan be at least one of a proprioceptive feedback device and a systemfeedback device. In case of a proprioceptive feedback device, feedbackcan be given in terms of relative limb position, grip type, temperature,surface condition, material condition like soft, slippery, rigid etc. Incase of a system feedback device, feedback can be given in terms ofsystem status, status of energy source etc. The feedback device can beat least one of a force feedback device, a tactile feedback device and atemperature feedback device.

According to a further object of the invention, there is provided amethod for controlling an electrically controllable limb device by acontrol unit, the limb device being in the form of an artificial limbfor a human or a robot limb, the limb device comprising a plurality ofactuators, the control unit comprising a first interface for connectingthe control unit to the limb device, the control unit comprising asecond interface for connecting the control unit to a data gatheringdevice comprising one or more sensing devices, the control unitcomprising a processing unit which is arranged for controlling the limbdevice at least based on data gathered by the data gathering device,wherein the control unit outputs one single control action step to theactuators of the limb device calculated by the processing unit based ona first data or data combination received from the data gatheringdevice, and the control unit outputs a plurality of control action stepsto the actuators of the limb device calculated by the processing unitbased on a second data or data combination received from the datagathering device, the second data or data combination being differentfrom the first data or data combination, the plurality of control actionsteps inducing a more complex automatic movement of the limb device thatthe one single control action step.

According to an advantageous embodiment of the invention myoelectriccontrol is used only for triggering, supervising and fine tuning of theautomatic operations of the control unit. In this way, myoelectricinformation is still used for control of the limb device, but more in abackground fashion, in order to release the user from too much burden inactivating several detailed movements of the limb device through itsmuscles.

According to a further object of the invention, there is provided acomputer program which is arranged for executing any of theaforementioned methods if the computer program is executed on aprocessor of the control unit controlling the limb device. The computerprogram can be stored on a storage medium, like a hard disk, a CD or anykind of semiconductor memories, like a memory stick.

In summary, the present invention comprises, among others, the followingaspects and further advantages:

A major improvement within the control is the use of the artificialvision in both closed loop automatic control of the prosthesis and theaugmented reality feedback to the user. The new control will allow theprosthesis to accomplish several tasks autonomously; therefore, decreasethe burden on the user and greatly improve the performance. Importantly,the user would still preserve her/his supervisory role by triggering,monitoring and correcting the performance.

The augmented reality feedback exploits the high bandwidth and fidelityof the artificial visual and human perception system, and overcomes thelimitations of the tactile feedback. The proposed scheme is notexcluding tactile and proprioceptive feedback coming from the sensorsintegrated in the hand; yet, the artificial vision conveys uniqueinformation during a prehension and preparation phases that cannot beprovided using the “built-in” position and force sensors.

A major improvement of the invention is that the reach/grasp functionscan be based on the integration of vision, perception, skills, and finetuning based on the tactile/force feedback.

We propose a novel control method in which some control steps areimplemented fully automatically, e.g., selection of grasp type and size,and orientation of the arm/hand, without the need for the userinvolvement. As a result, the control is simplified, and the cognitiveburden to the user is therefore minimized. In addition, we close thecontrol loop by providing rich visual information about the currentstate of the prosthesis to the user.

In this invention, we propose a system for the closed loop control ofgrasping and orientation in the arm/hand prosthesis. The control ofgrasping includes the possibility of automatic selection of the grasptype and aperture size that are appropriate to grasp the target object.The control of reaching refers to automatic adjustment of the hand jointangles so that the hand is properly oriented to implement the mostappropriate grasp.

We also propose to close the control loop by providing to the user avisual feedback about the state of the prosthesis (e.g., joint angles,grasping force). Using augmented reality, we can convey rich, highfidelity information with an information bandwidth that is significantlyhigher compared to a conventional, tactile feedback.

A major improvement of the proposed invention is to enrich theartificial controller by providing it with some extra sources ofinformation in addition to the classically used muscle electricalactivity. In this way, the artificial controller becomes capable ofmaking autonomous decisions and operating the hand automatically andwith minimal involvement of the user. The controller analyses theproperties of the target object and automatically selects the grasp typeand size, and adjusts the orientation of the hand to appropriately graspthe object.

The system employs stereovision implemented using special augmentedreality glasses with two cameras to obtain a 3D structure (geometricalmodel) of the scene and the target object. Since the controller has theinformation about the geometry of the target object, it can autonomouslydetermine the most appropriate way to grasp the object. Therefore, thecontroller automatically selects the grasp type and the size of the handopening. Next, an inertial sensor is mounted onto the artificial handproviding information about the hand orientation. This can be comparedto the orientation of the object, which is derived from the stereovisionprocessing, and used to 1) readjust the hand based on the direction(object side) the hand approaches the object, and/or 2) to control thehand orientation by adjusting the wrist (and elbow) joints of theprosthesis. Finally, the augmented reality glasses are used to provideto the user a rich visual feedback about the current state of theprosthesis. This feedback is projected into the scene as a virtualobject seamlessly integrating with the real environment. The feedbackprovides information about the position and orientation of theprosthesis, and also about the grasping force generated by the hand.This information can be used by the user to supervise the operation ofthe artificial controller, and when needed, to fine tune or correct theeventual mistakes made by the automatic control system. A simplemyoelectric control interface is used to trigger the automaticcontroller and, when needed, to manually control the prosthesis in aclosed loop by using the feedback.

According to an exemplary practical application, the user wearsaugmented reality glasses, inertial sensor(s) is placed on theprosthesis and myoelectric control interface is in place. The user looksat the target object and triggers the hand opening by using a simplemyoelectric command, e.g., finger extension. The hand automaticallypreshapes, i.e., the grasp type and size are automatically selected byanalyzing the geometrical properties of the target object. A virtualobject representing the prosthesis aperture and orientation is projectedinto the real scene just next to the object to be grasped. By comparingthe properties of the virtual object with the target object, the usercan fine tune or correct the grasp using myoelectric control. Using theinformation from the inertial sensor, the controller reacts andautomatically readjusts the grasp depending on how the user approachesthe object (e.g., from which side). If more proximal joints areavailable (wrist and elbow), the controller adjusts the orientation ofthe prosthesis during reaching in order to properly position the handfor the grasp. When the hand is positioned around the object, the usergives a command for the hand to close. A virtual object representing thecontact force is projected into the real scene just next to the objectto be grasped. The user controls the grasping force using myoelecticcontrol and augmented reality force feedback information.

As mentioned, the overall control paradigm is novel. Conventionalcontrol methods rely on the user to make all the decisions and sendcontrol signals to implement those decisions. In our approach, thecontroller is more “powerful” and brings autonomous operation to theartificial hand. The user merely triggers, supervises and corrects thesystem. This decreases the cognitive burden from the user.

The application of stereovision for the automatic control of the limbdevice, e.g. of grasping and reaching with the hand/arm prosthesis, isnovel. To coordinate the two separate control systems, we employ asimple myoelectric interface (two channels) to decode the userintentions and inertial sensors to assess how he/she moves theprosthesis.

The use of augmented reality (AR) to provide the rich visual feedbackabout the current state of the limb device is novel. AR was not appliedbefore in the context of the closed loop control of limb devices, e.g.hand/arm prosthesis. The latter represents a fully wearable and mobileapplication of AR.

The AR feedback is used not only to provide the status of the prosthesisvariables, but also to establish the communication between theartificial controller and the user. For example, the system indicates tothe user the object that is currently selected by the system as thetarget for grasping by marking it using a flickering red patch.

The application of inertial measurement units to measure the prosthesisorientation is novel, and the integration of inertial and stereovisiondata for the control of hand/arm configuration/orientation.

The proposed combination and integration of different technologies(stereovision, augmented reality, inertial units and myoelectriccontrol) into a complete solution for the closed loop control of thelimb device is a novel and unique approach in this field.

The control of grasping and reaching (orientation) for electricallypowered hand/arm prosthesis is a complex task. Myoelectric control isintuitive and simple to implement, but it is effective only when appliedto the simplest systems, e.g. single degree of freedom. If there aremore degrees of freedom available, they have to be controlledsequentially, one by one. For example, if the prosthesis implementsseveral grasp types, the user first has to generate muscle activity toselect the active grasp and then he/she generates muscle activity tocontrol the hand opening/closing. If there is an active wrist, the userhas to change the control mode from hand opening/closing to wrist jointcontrol in order to adjust the orientation. In order to accomplish this,the user needs to know the system and the control interface very well,which means that he/she has to go through a tedious and long trainingprocess. In addition, since the user has to go through a sequence ofsteps, controlling every single detail in this sequence, the wholeprocess can represent a significant cognitive burden.

In this invention, we propose a system that implements the control ofgrasping and hand orientation in a fully automatic way (transparently tothe user). The user has to look at the target object, which is doneanyway, and trigger the system, and the artificial controller analysesthe object properties and autonomously decides and controls the grasptype, size and hand/arm orientation. Since this is done fullyautomatically, the cognitive burden is truly minimized.

In this invention, we propose the use of augmented reality (AR) providedthrough an AR glasses to implement a rich visual feedback to the user.Vision has significantly higher bandwidth compared to the tactile sense.The visual feedback can convey rich, high fidelity information in a waywhich is easy to understand. At the same time, the user needs a minimaltraining to utilize the feedback successfully. This is contrary toconventional feedback interfaces that are limited to relatively simpleinformation and where a significant training is needed for the user tolearn how to decode the information communicated by the system. Since werely on AR, the feedback is naturally embedded into the real scene. Itis therefore nonintrusive, and it can be positioned in a way thatminimizes the interference with the rest of the scene. For example, inour current prototype, a virtual feedback object is placed next to thetarget object to which the user is looking, minimally blocking the otheraspects of the scene.

As pointed out in the following description, our system and method ishighly modular. The components stereovision control, augmented realityfeedback, inertial sensing are self-contained units that can be employedin combinations, using sensor fusion, or individually. This loosecoupling between the modules decreases the level of risk: a single part,a subset of components, or the system as a whole can grow into asuccessful product.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is now described by some examples using drawings.

The drawings show in

FIG. 1—an overview of a system of the invention, and

FIG. 2—a diagram showing a first aspect of the system operation, and

FIG. 3—the overall design of the system, and.

FIG. 4—a diagram showing a second aspect of the system operation.

DETAILED DESCRIPTION THE INVENTION

The invention that we have developed implements both feedforward andfeedback pathways providing the user with the possibility to control theprosthesis in a closed loop. We use computer vision in stereoconfiguration to emulate the aforementioned process of grasp planning inan able bodied human. The computer vision methods are employed toanalyze the scene, segment out the target object, analyze its 3Dproperties and automatically select the grasp type and size appropriatefor the object. To implement stereovision, we use special augmentedreality (AR) glasses that are equipped with a stereo camera pair. At thesame time, we use the glasses to provide an advanced visual feedback (inthe form of AR) about the current state of the prosthesis. Since ARobjects are integrated into the real scene, this feedback is simple,intuitive and effective, and at the same time, it can convey rich andhigh fidelity information, since the visual sense has a large bandwidth.Finally, we use inertial sensors to assess the current orientation ofthe hand and we fuse this information with a 3D model of the targetscene (available from the stereo vision engine).

The resulting system implements a high level automatic control ofgrasping and provides a rich feedback to the user by fusing severalinformation sources. In this scenario, the myoelectric control isreduced to triggering, supervising and fine tuning of the automaticoperations of the system. As a result, a significant part of the burdenfrom the user is relocated to the artificial control system.Importantly, this approach is powerful enough to control systems ofalmost arbitrary complexity, e.g. from an artificial hand to a full arm.The proposed system and method is an example for the use of sensorfusion in the control of prosthesis, which is the approach recommendedas the way to go in the future developments within this field.

We demonstrate a system of the invention by using a transradialprosthesis, i.e. artificial hand, as the example limb device to becontrolled. The hardware includes the following components (see FIG. 1):transradial prosthesis 2, augmented reality glasses 3, 4, EMG inputchannels 6, 18, inertial measurement unit (IMU) 5, control unit 1, whichcan be e.g. a standard laptop. In the sequel, we described thecomponents individually.

Transradial Prosthesis 2

We use one of the modern, flexible prosthesis, e.g. SmarthHand from theBiorobotics lab of Scuola Superiore Sant'Anna, Italy. The system hasindependently controllable fingers, and this can be exploited toimplement several grasp types (palmar, pinch, lateral and spherical) andto control the aperture size. The prosthesis is equipped with positionand force sensors; therefore, we can read the current finger positionsand contact force. This information is used to provide the feedback tothe user.

EMG Input Channels 6, 18

The EMG input channels 6, 18 comprise of two channels of bipolar EMG.Each channel has an amplifier 18. The signals of the amplifiers 18 arefed to an analogue to digital converter, e.g. an A/D acquisition cardmounted in the standard laptop. The channels will be placed on theresidual limb over the muscles that flex and extend the fingers, i.e.open/close the hand. This component is used to implement a simple twochannel myoelectric control that is used to trigger and fine tune theautomatic operation of the system.

Augmented Reality (AR) Glasses 3, 4

We use the model Vuzix iWear920AR. The model has two cameras in the formof a stereovision camera arrangement 4 that are built into the right andleft “glass”. Just behind the cameras, there is a visual output device 3in the form of two video panels projecting the video streams from thecameras. The video panels implement a stereoscopic 3D video, meaningthat the wearer (user) looking into the panels receives a 3D picture ofthe scene in front of him. The video streams captured by the two camerasare inputs for a stereo vision algorithm in the control unit 1, which isused to build the 3D model of the scene, analyze the target object anddecide grasp type and size. Before projecting the captured streams backto the panels, the video can be processed, and virtual objects can beintegrated into the scene. This method is used to provide the ARfeedback to the user.

Inertial Measurement Unit 5

This is a cluster of sensors, including accelerometers, gyroscopes andmagnetometers (XSens M420). The information from these sources iscombined, and the unit estimates a 3D orientation of the object to whichit is attached. This is used to provide the feedback about the currentorientation of hand, and also to control the hand during reaching.

Control Unit 1

Any computer, e.g. a standard laptop, can be used to implement thesignal processing and control algorithms. The control unit 1 receivesthe inputs from the other components, processes the data and sends thecontrol commands.

For example, the control unit 1 can also be implemented as en embeddedsystem, e.g. by using a microprocessor or microcomputer for the signalprocessing and control algorithms. The components can be miniaturizedand integrated into the hand prosthesis 2 itself. For example,myoelectric hands already have built-in EMG amplifier and electrodes.Also, a specialized processing unit (embedded, small factor system) witha powerful DSP processor can be used. FIG. 1 is a schematicrepresentation of the system with its processing modules and data flow.

In the exemplary embodiment the user operates a transradial prosthesis2. The EMG amplifier 18 provides two channels of bipolar EMG taken fromthe finger flexor and extensor muscle groups. This is a simplemyoelectric control interface for the triggering of the automaticcontrol. The user wears augmented reality (AR) glasses 3, 4 providingthe video streams from the stereo camera pair 4, which is the input forthe computer vision algorithm, and visual feedback to the user in theform of virtual objects embedded into the real scene. The inertialmeasurement unit 5 is placed onto the artificial hand 2 to measure its3D orientation.

The control unit 1 comprises of a computer, e. g. in the form of amicroprocessor or microcontroller 10 and a memory 11 where a controlprogram in the form of a computer program is stored. The control unitfurther comprises several interfaces 7, 8, 9 which connect the controlunit 1 to the external elements. A first interface 1, e. g. in the formof a serial data interface, e. g. a RS-232 interface, connects thecontrol unit 1 to the hand prosthesis 2 or any other limb device. Asecond interface 7, e. g. in the form of an USB hub, connects thecontrol unit 1 with any of the data input sources 4, 5, 6 which togetherform the data gathering device. A third interface 9, e. g. in the formof a video out interface, connects the control unit 1 to the stereodisplays 3 of the AR glasses, or to any other video output device.

The control unit runs several control processes, e. g. in the form ofsoftware processes executed by the computer 10. A first process 12 is amyoelectric control module which receives the digital EMG signals andoutputs signals derived from the input signals to a control block 16which is a finite state machine. Upon the signals of the myoelectriccontrol module 12 the finite state machine outputs trigger signals whichare marked in FIG. 1 by the dotted arrows. E. g. the finite statemachine 16 could trigger another control module which is a hand controlmodule 17. The hand control module 17 outputs control action stepsthrough the first interface 8 to the actuators of the hand prosthesis 2.

Another control module is a computer vision module 13 which comprises anobject 3D model 14 and an augmented reality box 15. The computer visionmodule 13 can be triggered by the finite state machine 16. The computervision module 13 receives signals from the elements 4, 5, 6 of the datagathering device through the second interface, e. g. it receives Eulerangles and stereo images. The computer vision module 13 outputs preshapecommands to the hand control module 17. Further, the computer visionmodule 13 outputs an augmented stereo/video stream through the thirdinterface 9 to the stereo displays 3.

Processing modules and data flow: The computer 10 implements signalacquisition and processing. The myoelectric control module 12 processesthe recorded EMG and detects the activity of the finger flexor andextensor muscles. The binary information about the muscle activitydrives the state transitions within the finite state machine 16implementing the overall control logic (explained in the text). Computervision module 13 receives the stereo images and reconstructs the 3Dgeometrical model of the scene and of the object that is the target forgrasping. Based on this and the information about the hand orientation(inertial measurement unit 5), the hand control module 17 decides thegrasp type and aperture size appropriate to grasp the object. Handcontrol module 17 sends the commands to the hand prosthesis 2 toimplement the desired grasp strategy. Computer vision module alsoreceives the sensor data from the hand prosthesis (position and force)and uses this to augment the video stream before it is re-projected tothe glasses to provide the visual feedback to the prosthesis user. Theprocessing is organized in a sequential manner and the modules areactivated (triggered) by the finite state machine.

FIG. 2 shows the system operation. There are e.g. eight steps 20, 21,22, 23, 24, 25, 26, 27 comprising the control loop of the system.

As can be seen in FIG. 2, the control loop is segmented in three phases,a first phase, a second phase and a third phase. In the first phase,which comprises steps 20, 21, 22, in step 20 the prosthesis is commandedto fully open. Then in step 21 the system acquires a stereo image pairand calculates the disparity map. Then, in step 22 it is expected thatthe user triggers the system by an extension of a finger.

The second phase comprises of steps 23, 24, 25. In step 23, an object isextracted and analysed in the 3D space. Then a step of sensory datafusion is executed. Then in step 24 an automatic grasp type selectionand prosthesis preshaping is executed. Proprioceptive AR feedback isgiven. Then in step 25 it is expected that the user triggers the system,e. g. by a flexion movement.

Finally, the third phase comprises of steps 26 and 27. In step 26 theprosthesis closes. The existing force is signalled by AR feedback. Thenin step 27 it is expected that the user triggers the system, e. g. by anextension of a finger.

In more detail, a target object selection can be performed by:

a) the user focuses on the target object,b) the computer vision module estimates the depth map and segments theobjects,c) the object closest to the center is selected as the target, and theuser sees the object marked by a flickering red color on the stereodisplays 3; the user issues the command for hand opening, and thisstarts the hand preshaping phase,d) a geometrical model of the object is constructed,e) a set of rules is used to select grasp type and size based on theestimated object properties, and the hand is commanded to preshape,f) a virtual object is embedded into the real scene (augmented reality)providing the information about the hand aperture size and orientation;the user places the hand close to the object and generates command forclosing,g) the hand closes around the object andh) a virtual bar is embedded into the real scene depicting the currentgrasping force (bar length and color).

The control loop operates sequentially, following the steps below:

1) Target Object Selection

The user observes the scene and “locks on” the object that he would liketo grasp. This is performed by orienting the head and thereby the ARglasses 3, 4 so that the desired object comes close to the center of thefield of view. The computer vision module continuously analyzes in softreal time the stereo video stream, estimates the 3D structure of thescene, establishing a depth map, and segments the objects. The objectthat is located most centrally is chosen by the system to be the targetfor grasping. To facilitate this process and make it transparent for theuser, the system “communicates” with the user using augmented reality:the object that is currently chosen by the system is marked by aflickering red color.

2) Hand Opening

When the user selects the object, she/he gives the command for the handto open by activating finger extensors. Note that this corresponds tothe muscle activity in able bodied subjects, i.e. fingerextensors/flexors are normally activated when opening/closing the hand.

3) Hand Preshaping

Computer vision module provides the 3D properties of the target object(position, orientation, and geometrical model. The objects are modeledas cylinders, boxes, spheres and lines, in case of thin and longobjects. This set of geometrical primitives covers the general shape ofthe most of the common daily life objects. The object features are inputfor a set of simple rules designed to emulate the cognitive processduring grasp selection in able bodied individuals. The rules haveIF-THEN structure: for example, if the object is cylindrical and wide,the palmar grasp is used, if the object is a thin box, lateral grasp isemployed etc. After the grasp type is selected, the aperture size isdetermined. The hand should open so that the aperture is wider than theobject, but not too wide as this can compromise the formation of astable grasp, e.g., the fingers could push out the object while tryingto close around. The appropriate aperture is calculated by adding asafety margin to the estimated object width. The hand is commanded topreshape according to the selected grasp strategy.

4) AR Feedback (Proprioception) with Manual/Automatic Adjustment ofAperture

The proprioceptive AR feedback is shown revealing to the user thecurrent hand aperture and orientation. The virtual box is embedded intothe real scene next to the target object and along the side from whichthe object is supposed to be grasped by the user. The size of thevirtual box is continuously updated to represent the current handaperture size, i.e., the box stretches and shrinks, while theorientation of the box follows the orientation of the hand, i.e., thebox rotates. Since the virtual box is placed next to the object, theuser can easily compare the current state of the hand with the actualproperties of the target object, evaluating the conditions for thestable grasp. Importantly, this can be done without focusing to the handand even while the hand is out of the user's field of view.

In the case when the hand aperture is not appropriate, e.g. wrongestimation from the computer vision module, it can be adjusted by usingsimple myoelectric control, i.e. to open/close the hand more, the useractivates finger extensor/flexor muscles. Therefore, by exploiting theAR feedback, the user can correct and/or fine tune the automaticdecisions of the artificial control system. At the same time, the systemcan recognize and react to the current activates of the user. Forexample, since the system “knows” the current orientation of the hand,it can detect that the user decided to approach the object from adifferent side with respect to the one suggested originally by thesystem. Therefore, the hand can be reshaped. The grasp size and eventype can be changed reactively.

5) Hand Closing

When the user estimates that the hand configuration is appropriate,she/he issues the command for closing. This is done by activating fingerflexor muscles, which again corresponds to the way these muscles areused in able bodied subjects.

6) AR Feedback (Force) with Manual Force Control

Once the contact with the object is detected, AR feedback of graspingforce is started. The bar indicating the current force appears next tothe object, e.g. a red bar. The bar length and color are proportional tothe force amplitude. This is direct and intuitive information about thevariable (force) that is almost completely unobservable under the usualconditions, i.e., a conventional prosthesis without feedback.

The system of the invention is modular and flexible. Most of thecomponents can be used as standalone functional modules. For example,the control based on stereo vision or the feedback via augmented realitycould be used on their own, without the integration with each other orthe other components.

Many aspects of the aforementioned control loop can be easily extendedto accommodate more sophisticated features. During the target objectselection, eye tracking can be used to determine the object that theuser is currently looking at. Importantly, the eye tracking algorithmmight be embedded into the glasses. For the hand preshaping, a generalhigh fidelity model (triangle mesh) of the target object can be usedinstead of the limited set of predefined geometrical primitives. The ARproprioceptive feedback can be implemented by using more sophisticatedgraphical representation, e.g., graphical model of the hand. The same istrue for the force feedback. During the approach phase, if the hand hasa wrist rotator, the orientation of the hand can be automaticallyreadjusted so that the hand grasps the object from the side (and/or atthe sites) that was selected previously during the grasp planning. Ifprosthesis is more complex, e.g. whole arm, the stereovision algorithmcan drive all the degrees of freedom during reaching and grasping.

An important step in stereo processing is the depth (disparity)estimation, which is the method to estimate a distance from the camerasfor some or all of the pixels in the image. The success of this step hasa major impact on the quality of all further image-processing phases. Anefficient Large-scale Stereo Matching (ELAS) algorithm is applied to agrayscale image pairs to determine the depth map. The 3D coordinates foreach pixel are then easily obtained by combining pixel disparityinformation and intrinsic camera properties (triangulation).

The scene analysis is started by identifying the surface on which theobjects are placed, i.e., a surface of the table. This base surface ismodelled geometrically as a simple plane. The target object located onthe base plane is modelled as a box, cylinder, line or sphere. Thesegeomet-geometrical primitives are determined by assuming a certain modeland fitting it through the point cloud (pixels with 3D coordinates)using RANSAC algorithm; the model with the best fit is adopted torepresent the shape of the object. This process is described below:

Plane: three support points in the point cloud are determined that havethe largest number of inliers (where an inlier is every point that has adistance to the plane of less than 0.3 cm).

Box: The method iteratively fits planar surfaces until it finds twoplanes with compatible orientation (i.e., they must intersect and forman angle that is close to 90°). Using the intersection line, directionsof these two planes and of the remaining faces are reconstructed and asemi closed 3D patch structure is formed.

Cylinder: The method searches for a set of three points that form acircle in the 3D space, which when extended to cylinder contains thelargest number of inliers (distance from surface less than 0.2 cm).

Line: A line is fitted through set of points using LSQ method and themost common length of all other lines that are perpendicular to it isdetermined.

To provide the AR feedback, a virtual box object is embedded into thereal scene. This is done by forming a 3D model of the AR box within theexisting point cloud, and then finding its visible faces from the users'perspective and back projecting them on the left and right 2D cameraimage planes. The end result is an augmented reality box object thatappears as a part of the real scene observed by the user.

In the following, the invention is further explained using FIGS. 3 and4.

Generally, the invention presents a method and a system for thesemi-autonomous closed loop control of a multi degree of freedom (DOF)upper limb device (ULD) for the assistance of reaching and grasping. Inparticular, the upper limb device can be a prosthesis (e.g., artificialrobotic hand or arm) or an orthosis (e.g., robotic exoskeleton). Insummary, the system integrates:

-   -   1) An artificial controller that combines the state of the art        technologies (e.g., computer vision, inertial sensors, augmented        reality, and myolectrical control) in order to perform a fully        autonomous operation of the ULD during typical reaching,        grasping and manipulation tasks (e.g., automatic preshaping of        the gripper, control of reaching and orientation of the        ULD/gripper).    -   2) Artificial feedback interface providing to the user the        missing proprioceptive and grasping force sensory information        from the ULD, thereby implementing the user-driven closed loop        control of the ULD.    -   3) Bilateral communication interface to support the shared        control between the user and the semi-autonomous controller. The        user supervises the automatic operation of the system using        feedback interface (controller=>user), and in any moment in        time, he/she can correct, fine tune or override the autonomous        operation using manual control (user=>controller).

The invention comprises a modular system and a method for thesemi-autonomous closed loop control of a multi-DOF upper limb device(ULD) for the assistance of reaching and grasping. The system comprisesat least fol-following components (see FIG. 3):

-   31: A set of sensors providing information about the current state    of the ULD and its interaction with the environment.-   32: A set of sensors providing high fidelity information about the    2D/3D spatial structure of the environment, from which the    properties of the object that is the target for reaching/grasping    (hereafter target object) can be estimated. This can be e.g. the    stereo cameras 4 of the AR glasses.-   33: A semi-autonomous programmable electronic controller (hereafter    controller) that fuses the data from the sensors 31 and 32 and based    on this automatically (without user intervention) controls grasping    and/or reaching functions of the ULD. The controller 33 can be the    control unit 1.-   34: Man-machine interface to capture and decode the commands from    the user. The commands are received by the controller 33 and used to    trigger, supervise, correct, fine tune and/or override the automatic    operation of the system.-   35: Feedback interface providing to the user online information    about the state of the ULD and/or controller.-   36: An upper limb device, i.e., the system to be controlled.

In one particular implementation, the ULD is an electrically controlledupper limb prosthesis or an upper limb robotic exoskeleton. ULD is amulti DOF mechanical system comprising an arbitrary number of actuatorsthat can be individually controlled and sensors measuring the devicestate and/or interaction with the environment.

In one particular implementation, the sensors 31 are embedded into theULD and measure the position of the ULD articulations (e.g., joinangles), interaction forces (e.g., grasping force), orientations of theULD segments (e.g., inertial measurement units) or any otherinternal/interaction variable of interest (e.g., temperature,vibration). Alternatively, the sensors can be placed externally onto theULD or they can be integrated into the controller 33 (see below). Theexternally placed sensors can be connected to the controller using anywired or wireless communication link.

In one particular implementation, the sensors 32 include any sensingdevice that can provide the information about the structure of theenvironment from which the properties of the target object can beestimated. This could be ultra sound sensors, laser scanners or activeimaging devices such as a time of flight camera and structured lightsensors (e.g., Kinect) and/or passive imagining devices such as singleor multiple video camera systems (e.g., stereo camera pair). Thesesensors provide 2D/3D image of the scene and the target object. Thesensors can be embedded into the ULD, mounted externally onto the ULD orthey can be worn by the user. Possibly, the sensors can be integratedinto the feedback interface 35 (see below). For example, a camera can beintegrated into the hand prosthesis or embedded within the frame of theaugmented reality (AR) glasses. The communication link between thecontroller and the sensors can be wired or wireless.

In one particular implementation, the controller 33 can be an embeddedreal time system integrated into the ULD or an external programmableelectronic device. The latter can be a custom made unit or a generalpurpose processing element such as a smartphone. In both cases, thecontroller can have integrated sensors (e.g., a smartphone with anaccelerometer or a camera) that can be used within the system as thesensors 31 or 32.

In one particular implementation, the controller 33 uses 2D/3Dinformation about the target object provided by the sensors 32 toautomatically preshape the gripper of the ULD so that its configurationis convenient for grasping the target object. For example, thecontroller estimates the properties (shape and size) of the targetobject from a 2D/3D image of the scene provided by an imaging device.Based on this, the controller selects a grasp type from a predefined setof implemented grasps and adjusts the size of the hand opening so thatthe object fits nicely into the hand. Also, the controller coulddirectly adjust individual finger joints taking into account detailedinformation about the object shape.

In another particular implementation, the controller 33 combines 2D/3Dinformation about the target object obtained from the sensors 32 and theinformation about the state of the ULD from the sensors 31 toautomatically control the reaching movement towards the object as wellas the orientation of the gripper. For example, the controller estimatesthe orientation of the target object from the acquired 3D image. Fromthe position and orientation sensors within the ULD, the controllerdetermines the current orientation of the ULD. Based on this, itcalculates the reference reaching trajectory and transports the ULD fromits current position to the vicinity of the target object. At the sametime, the controller adjusts the joints of the ULD so that at the end ofthe reaching movement the gripper is positioned conveniently to graspthe target object. The control of reaching and orientation accommodatesthe active joints of the ULD. For example, in the case of a roboticprosthetic hand the wrist joint is controlled (e.g., orientation of thegripper), while the shoulder, elbow and wrist are controlled in the caseof a full arm prosthesis (e.g., both reaching and gripper orientation).

In yet another implementation, the controller 33 uses the informationfrom sensors 31 and 32 to make the ULD dynamically respond to useractions and intentions. For example, by monitoring the ULD orientations(through sensors 31) and by knowing the orientation and shape of thetarget object, the controller can detect the side the user isapproaching the target (e.g., short or long side) and reconfigure thegripper accordingly (e.g., smaller or larger opening).

In one particular implementation, the controller 33 selects the targetobject from a set of detected objects as the one that is the closest tothe center of the 2D/3D image provided by the imaging device. In anotherparticular implementation, a miniature eye tracking device is integratedinto the system. The controller then selects the target object as theobject to which the user points his/her gaze.

In one particular implementation, the man machine interface 34 isimplemented using myoelectric control and/or other sensors. It acquiresthe command signals from the user and converts them into system actions.The user manual commands have the highest priority and always overridethe decisions of the controller. Put differently, the user can take overthe command of the system in any moment, and manual control hasprecedence over the automatic control. In principle, any of thecurrently available technologies for implementing the communicationbetween man and machine can be exploited to implement this component(e.g., voice, brain machine interface). In one possible implementation,inertial measurement units can be used to detect the movement anddetermine the user intensions in parallel to the conventionally usedmyoelectric control, making the overall man-machine interface morerobust.

In one particular implementation, the feedback interface 35 can be awearable visual display. For example, in the most basic version, thiscan be a display attached to the prosthesis. Alternatively, for the moresophisticated implementation and better performance, this can be asemitransparent or nontransparent augmented reality (AR) glasses orcontact lenses designed either for monocular or stereoscopic viewing.These devices can also integrate a variety of additional sensors (e.g.,inertial sensors, global positioning sensors, cameras), and these can bealso used as the sensors 32 providing multimodal information about theenvironment. AR is used to transmit the visual feedback to the userabout the current state of the ULD or the controller. A virtual objectembedded into the real scene can convey, for example, the configurationof the ULD (artificial proprioception) or interaction forces (artificialforce feedback) or the selected grasp type andthe plannedfinalorientation of the ULD (controller state feedback). The visual feedbackvirtually places the ULD within the field of view of the user, even whenthe physical device is outside of his/her field of view. Therefore, itemulates the biological condition and makes the control more efficient,since the sensor data about the device state are always available (ULDstate feedback). Also, the user has an insight into the operation of theautomatic control (controller state feedback), which he/she can use tosupervise the autonomous operation and revert to manual control wheneverneeded. Furthermore, the feedback can be used to provide the informationthat is normally not directly observable (e.g., force, temperature) ormore precise/detailed information about the device state (e.g.,amplified small finger movements during a fine manipulation task).Virtual object(s) conveying the feedback can have different forms, fromsimple to very sophisticated. For example, a virtual bar plot placed inthe peripheral visual field of the user can represent the currentopening or grasping force of the ULD gripper. Or, the grasp typeautomatically selected by the controller can be represented to the usereven before the gripper assumes the selected shape. Alternatively, ageometrical model of the full hand can be drawn in front of the targetobject showing directly the relationship between the object and thehand.

In another particular implementation, the feedback interface 35 can beimplemented using some of the sensory substitution methods, such aselectro- and/or vibrotactile stimulation or any other form of tactilestimulation or in general any other method for the sensory restoration(e.g., sound feedback). This implementation of the feedback interfacecan be used alone or in combination with the aforementioned AR feedback.

Methods Description

The method for the semi-automatic control of the ULD comprises severalreal time processing modules implementing a sequence of steps (see FIG.4):

-   41: Data Acquisition module collects the data about the ULD state    and 2D/3D structure of the environment from the sensors 31 and 32,    respectively.-   42: Artificial Perception module analyzes the structural data,    identifies the target object and determines the properties of the    target object that are relevant for implementing automatic reaching    and grasping with the ULD.-   43: Sensor Fusion module combines the outputs of the Artificial    Perception module and the information about the ULD state to    determine additional information for controlling automatic reaching    and grasping.-   44: Cognitive Processing module uses the outputs from the Artificial    Perception and Sensor Fusion modules to autonomously decide    appropriate strategies to reach for and grasp the target object.-   45: ULD Control module sends commands to the ULD to implement the    strategies selected by the Cognitive Processing module (automatic    control) or to respond to the user commands (manual control).-   46: User Feedback module receives the data about the ULD state (Data    Acquisition), target object (Artificial Perception), and automatic    operation of the system (Cognitive Processing) and based on this    provides visual feedback (augmented reality) to the user.-   47: User Command module detects and decodes user commands and    intentions, and based on this, triggers the automatic control and/or    directly commands the ULD (manual control).

The method steps can be processed by a computer program executed on aprocessor of the control unit 1 or the controller 33.

In one particular implementation, Artificial Perception module relies onthe use of the state-of-the-art computer vision algorithms for analyzing2D/3D imaging data, including for example various stereo processingmethods and different clusterization, classification and featureextraction techniques. Some of the possible choices are Scale-InvariantFeature Transform (SIFT), Random Sample Consensus (RANSAC), SimultaneousLocalization and Mapping (SLAM), and Kinect Fusion. The outcome of thisprocessing is a geometrical model of the environment, including theobjects that might be potential grasping targets. In the next step, oneof the objects is selected as the target, and a geometrical model of theobject is constructed in absolute or relative coordinates. The objectmodel provides information about the physical object properties (e.g.,shape and size).

In one particular implementation, Sensor Fusion module compares thegeometrical object model and information about the configuration of theULD to determine the relative orientation of the ULD gripper withrespect to the object. It can employ prediction and filtering algorithms(e.g., adaptive Kalman Filtering) to estimate the absolute or relativepose of the user and/or prosthesis in 3D space.

In one particular implementation, Cognitive Processing module uses a setof rules to select an appropriate configuration and/or orientation ofthe ULD gripper from a predefined set of possible configurations and/ororientations. These rules map the information about the physicalproperties and/or pose of the target object to the appropriatepreshape/orientation strategy, and they can be hand crafted or computergenerated (machine learning). The rules can be explicit (“IF-THEN”,fuzzy networks, inductive learning) or implicit (e.g., neural networks).

In another particular implementation, Cognitive Processing moduledetermines the appropriate configuration/orientation of the ULD gripperfrom the continuous space of all physically possibleconfigurations/orientations. The module also calculates the trajectory(i.e., a sequence of transient configurations/orientations) to beimplemented in order for the ULD to attain the determined finalconfiguration/orientation. This processing can be performed by using themethods of “visual servoing”.

In one particular implementation, User Feedback module constructsgraphical objects that should convey visual feedback to the user. Thoseobjects are then projected to a wearable display device for thepresentation to the user. This processing is based on the conventionalcomputer vision methods (e.g., computer geometry and linear algebra).

In another particular implementation, the feedback is delivered throughtactile stimulation and in this case the User Feedback module implementsthe mapping from the variables of interest to the corresponding hapticstimulation patterns. Any of the tested coding schemes available in theliterature on sensory substitution or original coding algorithmsdesigned specifically for this application can be implemented.

In one particular implementation, User Command module employsconventional myoelectric control. Muscle activity is recorded usingelectromyography (EMG) and simple triggering is employed to activate theautonomous control of ULD. To implement manual control overriding theautomatic operation, proportional myoelectric control can be used. Theposition, velocity and/or grasping force of the ULD can be made directlyproportional to the recorded muscle activity (electromyography). Muscleco-activation is used to change the control mode.

In another particular implementation, User Command module implements anyof the state-of-the-art pattern recognition algorithms for myoelectriccontrol, or some of the latest generation methods for simultaneous andproportional control of multi-degree of freedom ULD.

In yet another particular implementation, User Command module can relyon some less conventional man-machine interfaces (e.g., voice,brain-machine interfaces etc.)

Example Implementations Example 1 (Hand Prosthesis with an ActiveImaging Device and Augmented Reality)

The user is equipped with the special glasses that are specificallydesigned for this application. The glasses have a see-through AR screensand miniature active imaging device embedded into the glass frames. Theimaging device provides a 3D structure of the scene in front of thesubject. All the components are small factor and they are integrated sothat these specialized glasses look very much like ordinary glasses.

The subject is an amputee and uses a dexterous prosthetic hand withindividually controllable fingers and active wrist joint allowing wristflexion/extension and pronation/supination movements. The hand isinstrumented with position and grasping force sensors, and also withinertial measurement units measuring the hand orientation in space. Thesensors are embedded within the hand. The programmable semi-automaticcontroller is also embedded into the hand electronics.

The glasses are connected to the controller via a wireless linktransmitting 3D information acquired from the imaging device to thecontroller and processed AR feedback information back to the glasses tobe drawn at the see through displays.

Triggering of the automatic operation and manual control duringsemi-automatic operation are implemented using simple myoelectricinterface comprising two channels of bipolar EMG.

Next, we present a hypothetical use case scenario. The user approachesthe table with a set of objects. He/she looks at the one he would liketo grasp and the system instantly provides visual feedback acknowledgingthat the object was recognized (e.g., the target object is marked). Thesubject reaches with the hand and at the same time triggers the hand toopen by briefly contracting his extensor muscles. While the hand travelstowards the object, the controller has already analyzed the 3Dinformation obtained from the imaging device, segmented the targetobject and determined its properties (shape and size). As a result, itcommands the hand to preshape into an appropriate grasp. Simultaneously,the controller continuously processes the inertial data estimating theorientation of the hand with respect to the target object. Based onthis, it automatically controls the two degrees of freedom in the wristjoint so that the hand is properly oriented for the grasp. When the handcomes close to the object, the user triggers closing and grasps theobject.

While holding, the user relies on manual control to manipulate theobject and control grasping force. The user controls the force byrelying on a simple AR feedback, a bar placed in the peripheral visiondepicting the current grasping force.

Note that the automatically controlled reach and grasp movement,operates very similarly to the natural movement. The user expresses thehigh level goal (grasp the object) and triggers the system, and the“hand” implements the rest of the steps.

Example 2 (Hand Prosthesis with Augmented Reality)

The user of a hand prosthesis wears standard commercially available ARglasses (e.g., Google glass project). The glasses are wirelesslyconnected to the controller embedded within the hand, which is operatedusing a conventional two channel myocontrol interface.

Controller receives position and force data from the embedded prosthesissensors and provides artificial proprioceptive and force feedbackthrough AR. Specifically, an ellipsis is drawn in the peripheral visualfield of the user by using AR glasses. The horizontal axis is madeproportional to the current hand aperture and vertical axis isproportional to the grasping force. Therefore, the user receivesinformation about the hand as the change in the shape of the ellipsis.This information is send by the controller to the glasses which projectthe image (ellipsis) to the user.

A hypothetical use case scenario could be as follows. The subjectapproaches the table with a set of objects. He/she starts reaching forthe target object and simultaneously starts preshaping the hand. This ispossible despite the fact that the subject in the moment he/she startsreaching does not see the hand. The subject however has the fullinformation about the configuration of the hand through the virtue ofthe provided AR feedback. After the grasp is established, the subjectcontrols the grasping force through the AR grasping force feedback.

Note that the reaching and grasping with this system simulates thesequence of events that is typical for the grasping of able bodiedsubjects. Namely, the amputee can start with the hand preshape while thehand is being transported to the target object, so that the hand isappropriately configured and ready for closing at the end of the reach.

Example 3 (Full Arm Prosthesis with an Active Imaging Device andAugmented Reality)

This implementation is very similar to the implementation presented inExample 1. The main difference is in the complexity of the prostheticsystem. In this case, the subject is equipped with a full arm prosthesiswhich has an active shoulder, elbow and wrist joints and independentlycontrollable fingers.

The use case scenario could look something like this. The user wants tograsp an object. He/she looks at the object and the system provides theacknowledgement that the object is recognized and selected as thetarget. The user triggers the system by providing a brief contractionusing one of his

1-21. (canceled)
 22. A system for controlling a hand prosthesiscomprising: electromyography (EMG) sensors configured to provide EMGsignals from a user's muscles; a three dimensional (3D) sensing deviceconfigured to generate 3D data including depth information of a targetviewed by the 3D sensing device; a control unit configured to receivesignals from the EMG sensors and the 3D sensing device, the control unitincluding: a myoelectric control module which processes recorded EMGsignals and detects activity of the user's muscles, a controllerreceiving information about muscle activity from the myoelectric controlmodule which drive state transitions within the controller to implementcontrol logic, a computer imaging module configured to receive 3D datafrom the 3D sensing device and reconstruct a 3D geometrical model of ascene which includes an object that is the target for grasping, and ahand control module configured to receive information from the computerimaging module and information about hand prosthesis orientation andgenerate a grasp type and an aperture size appropriate to grasp theobject, the hand control module configured to send commands to the handprosthesis to implement a desired strategy; and a feedback unitconfigured to receive information from the computer imaging module andproviding feedback to the user.
 23. The system for controlling a handprosthesis according to claim 22, further comprising an inertialmeasurement unit attachable to the hand prosthesis which generates 3Dposition and orientation information of the prosthesis, the control unitfurther configured to receive the 3D position and orientationinformation of the prosthesis.
 24. The system for controlling a handprosthesis according to claim 22, wherein the EMG sensors are configuredsuch that EMG signals are taken from finger flexor and extensor musclegroups of a user.
 25. The system for controlling a hand prosthesisaccording to claim 22, further comprising an artificial limb to whichthe hand prosthesis is attached, wherein the EMG sensors are configuredto obtain EMG signals from functional muscles around a shoulder of theuser.
 26. The system for controlling a hand prosthesis according toclaim 22, wherein the feedback unit includes a visual output deviceoutputting visual data showing positions of joints in the handprosthesis.
 27. The system for controlling a hand prosthesis accordingto claim 22, wherein the computer imaging module is also configured toreceive sensor data from the hand prosthesis in the form of position andforce data and use this data to augment the visual feedback to the user.28. The system for controlling a hand prosthesis according to claim 22,further comprising a stereo vision device configured as AugmentedReality (AR) glasses wearable by a user and controlled by the computerimaging module to display a 3D picture of a scene in front of the user.29. The system for controlling a hand prosthesis according to claim 28,wherein the computer vision module is configured to output a markingsignal in a video stream based upon an automatic selection by thecontrol unit of an object which is to be grasped, the marking signalmarking the selected object in the 3D picture of the scene in front ofthe user.
 30. The system for controlling a hand prosthesis according toclaim 22, wherein processing in the control unit is organized in asequential manner and the modules are activated (triggered) by a finitestate machine.
 31. The system for controlling a hand prosthesisaccording to claim 22, wherein the 3D sensing device is a 3D-scanningdevice.
 32. The system for controlling a hand prosthesis according toclaim 31, wherein the 3D-scanning device is selected from the groupconsisting of a stereovision camera device, a laser scanning device, aradar device, and a white-light scanning device.